Skip to content

Updated credentials rotation topic#1004

Open
n-boshnakov wants to merge 2 commits into
gardener:masterfrom
n-boshnakov:update-credentials-rotation-documentation
Open

Updated credentials rotation topic#1004
n-boshnakov wants to merge 2 commits into
gardener:masterfrom
n-boshnakov:update-credentials-rotation-documentation

Conversation

@n-boshnakov

@n-boshnakov n-boshnakov commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

How to categorize this PR?

/kind enhancement

What this PR does / why we need it:
This PR updates the Credentials Rotation page to better reflect the user needs.

Which issue(s) this PR fixes:
Part of #1002

Special notes for your reviewer:

Summary by CodeRabbit

  • Documentation
    • Reorganized the Credential Rotation documentation with clearer coverage of infrastructure and shoot credentials.
    • Added a scope and ownership overview for credential classes.
    • Documented the two-phase rotation model and automatic rotation behavior.
    • Clarified which credentials require action between rotation phases.
    • Updated guidance for infrastructure credentials and linked to detailed rotation procedures.

@n-boshnakov
n-boshnakov requested a review from a team as a code owner June 17, 2026 10:59
@gardener-prow gardener-prow Bot added the kind/enhancement Enhancement, improvement, extension label Jun 17, 2026
@netlify

netlify Bot commented Jun 17, 2026

Copy link
Copy Markdown

Deploy Preview for gardener-docs ready!

Name Link
🔨 Latest commit 36ac829
🔍 Latest deploy log https://app.netlify.com/projects/gardener-docs/deploys/6a5a55fb284b7300084a592f
😎 Deploy Preview https://deploy-preview-1004--gardener-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The credential rotation documentation now classifies credential types, links detailed procedures, explains two-phase and automatic rotation, and documents infrastructure credential storage and usage.

Changes

Credential Rotation Documentation

Layer / File(s) Summary
Credential classification and overview
hugo/content/docs/getting-started/features/credential-rotation.md
The overview distinguishes infrastructure and shoot credentials by scope and ownership and links to detailed rotation guides.
Rotation phases and automation
hugo/content/docs/getting-started/features/credential-rotation.md
The page replaces command-based phase instructions with the two-phase model and adds automatic rotation behavior and required user actions.
Infrastructure credential guidance
hugo/content/docs/getting-started/features/credential-rotation.md
The infrastructure section documents project-scoped Secrets, CredentialsBindings, rotation procedures, and infrastructure account constraints.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

Possibly related PRs

  • gardener/documentation#822 — Covers automated and maintenance-window credential rotation, which relates to the new automatic rotation section.

Suggested reviewers: donistz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: a rewrite/update of the Credentials Rotation documentation.
Description check ✅ Passed The description follows the template and fills the main sections, including categorization, purpose, issue reference, and reviewer notes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gardener-prow gardener-prow Bot added cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 17, 2026
@gardener-prow

gardener-prow Bot commented Jun 17, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dimitar-kostadinov for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@donistz

donistz commented Jun 17, 2026

Copy link
Copy Markdown

Core Problems in the Documentation

1. Conceptual overload without a simple mental model

The doc immediately introduces:

  • many credential types (CAs, SSH, ETCD, service accounts, observability, cloud keys),
  • two ownership models,
  • and a two‑phase rotation model

➡️ But it never gives a simple “big picture” first.

Impact: Stakeholders cannot answer basic questions like:

  • “What exactly am I rotating?”
  • “Why would I need this?”
  • “Which path applies to me?”

✅ What’s missing:

  • A 1-paragraph explanation of the problem rotation solves
  • A diagram or flow (who rotates what and when)

2. Two completely different processes are mixed together

The doc contains two fundamentally different workflows:

A. Gardener-managed credentials

  • rotated via annotations
  • two-phase model

B. User-provided credentials (cloud keys)

  • rotated by updating secrets
  • no phases
  • async reconciliation

➡️ These are described in the same flow, with weak separation.

Impact: Users blend the models incorrectly, e.g.:

  • Expect phases for cloud credentials ❌
  • Expect annotations for cloud credentials ❌
  • Don’t understand why one is manual vs automatic ❌

✅ What’s missing:

  • A hard separation with decision logic, e.g.:
START HERE:
Are you rotating cloud provider credentials?
 → Yes → Follow process A
 → No → Follow process B

3. The “two-phase rotation” is explained abstractly, not operationally

The documentation says:

  • “prepare phase”
  • “complete phase”
  • “update clients in between”

But it does NOT clearly answer:

  • Who are the clients?
  • What exactly must be updated?
  • How do I know I’m ready for “complete”?
  • What breaks if I don’t?

➡️ It explains mechanics, not usage.

Impact: Stakeholders don’t trust the process because:

  • It feels risky
  • It lacks operational checkpoints

✅ What’s missing:

  • A real scenario, e.g.:

“If you rotate the CA, kubeconfigs must be redistributed before completing.”


4. Too much Kubernetes-native detail, not enough user intent

Examples:

kubectl annotate shoot ...
.status.credentials.rotation
.spec.maintenance.autoRotation.credentials

➡️ These are implementation details, not user guidance.

Impact:

  • Non-expert users get lost
  • Even experts must mentally reconstruct the workflow

✅ What’s missing:

  • Step-by-step flows like:
    1. Trigger rotation
    2. Wait for condition X
    3. Update Y
    4. Verify Z
    5. Complete rotation

5. Ambiguous ownership messaging

Table says:

Class | Who rotates -- | -- User-provided | You Gardener-managed | You

➡️ Both say “You” → this is misleading.

Impact: Users ask:

  • “What exactly does Gardener automate then?”

✅ Reality (not clearly expressed):

  • Gardener executes rotation
  • User controls timing and readiness

✅ What’s missing:

  • Clarification like:

“Gardener performs the mechanics, but you are responsible for safe rollout decisions.”


6. No clear failure modes or risks

The doc never clearly explains:

  • What happens if you:
    • don’t update clients before “complete”
    • delete credentials too early
    • rotate in wrong order

➡️ This is critical for operational confidence.

Impact:

  • Stakeholders hesitate to use the feature
  • Or worse → misuse it

7. User-provided credentials section is underspecified

It says:

  • Update secret
  • Wait for reconciliation
  • Delete old keys

But lacks:

  • timeline expectations
  • verification steps
  • multi-shoot implications

➡️ Especially weak for shared credentials.


8. Repetition without clarity improvement

The update repeats ideas like:

  • “prepare phase adds new credentials”
  • “complete phase removes old credentials”

➡️ But repetition ≠ clarity

Impact:

  • The doc is longer but not clearer
  • Key insights are still missing

9. No concrete examples

There are zero real-world scenarios, such as:

  • Rotating CA → kubeconfigs impacted
  • Rotating SSH key → node access impact
  • Rotating ETCD encryption → no client action

➡️ All credentials are treated generically, but they behave very differently.


🧠 Why Stakeholders Still Don’t Understand It

1. They think in tasks, not in systems

They want:

“I need to rotate cloud credentials safely — what do I do?”

But the doc answers:

“Here is how credential systems work internally”


2. They need decision guidance, not reference text

Missing:

  • decision trees
  • “when to use what”
  • “which applies to me”

3. The doc assumes deep Gardener/Kubernetes knowledge

Concepts like:

  • Shoot
  • reconciliation
  • maintenance window

are not re-explained.


4. It lacks operational confidence signals

No:

  • checklists
  • validation steps
  • “safe to proceed” markers

✅ What Would Fix It

1. Add a decision entry point

Which credentials do you want to rotate?
[ ] Cloud provider credentials → go here
[ ] Cluster internal credentials → go here

2. Split into two completely separate guides

  • Guide A: Cloud credentials (simple, linear)
  • Guide B: Gardener-managed credentials (phased, more complex)

3. Add worked examples

At least:

  • CA rotation
  • SSH key rotation
  • Cloud credential rotation

4. Add checklists

Example:

Before completing rotation:

  • All kubeconfigs updated
  • All clients using new CA
  • Verification successful

5. Add a diagram

Visualizing:

  • prepare → coexistence → complete
  • or user vs gardener responsibilities

🟢 Bottom line

The documentation improved accuracy, but not usability.

👉 The real issue is not missing information
👉 It’s missing structure, decision guidance, and operational clarity

That’s why stakeholders still say:

“I read it, but I don’t know what to actually do.”

@JordanJordanov
JordanJordanov requested a review from donistz June 26, 2026 05:32
@gardener-prow gardener-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 16, 2026
@n-boshnakov
n-boshnakov force-pushed the update-credentials-rotation-documentation branch from ec76ecf to 36ac829 Compare July 17, 2026 16:19
@gardener-prow gardener-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 17, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hugo/content/docs/getting-started/features/credential-rotation.md`:
- Around line 49-57: Update the “Automatic Rotation” section to explicitly list
the OpenVPN TLS auth key as a credential that cannot be rotated manually or
automatically. Distinguish it from certificate authorities and the
ServiceAccount signing key, which require manual rotation between phases, and
preserve the existing automatic-rotation list.
- Around line 20-25: Update the credential responsibility table and surrounding
explanation to distinguish user actions from Gardener-managed rotation
mechanics: users choose timing, update dependent clients, and trigger
completion, while Gardener creates, distributes, and invalidates Shoot
credentials. Add a concise decision guide explaining when to use the
infrastructure-credentials workflow versus the Shoot-credentials workflow,
anchored to the existing “Infrastructure credentials” and “Shoot credentials”
sections.
- Around line 61-68: Expand the credential-rotation guidance around “When you
rotate these credentials” with actionable verification: specify how to confirm
every referencing Shoot has reconciled successfully, the expected timing or
polling interval, and the status conditions to check. Add failure handling for
an individual Shoot, including investigating or remediating it and keeping the
old provider keys active until all shared-Shoot reconciliations succeed.
- Around line 31-43: Expand the “Two-Phase Rotation Model” section with concrete
readiness checks before completing rotation, including how to verify all CA and
kubeconfig consumers use the new credentials. Describe the access failures
caused by completing with a missed client and document the recovery path for a
failed completion, using the existing rotation status reference where
applicable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: ab21d948-05ad-4052-9a21-abf5161a40ca

📥 Commits

Reviewing files that changed from the base of the PR and between cc262c0 and 36ac829.

📒 Files selected for processing (1)
  • hugo/content/docs/getting-started/features/credential-rotation.md

Comment on lines +20 to +25
| Class | Examples | Scope | Who rotates |
|---|---|---|---|
| **Infrastructure credentials** | Cloud provider keys (AWS, Azure, GCP, OpenStack) | Project-scoped — shared across Shoots via `CredentialsBinding` | You |
| **Shoot credentials** | CAs, SSH key pair, ETCD encryption key, ServiceAccount signing key, observability passwords | Per-Shoot — generated and managed by Gardener | You, via `kubectl annotate` operations on the Shoot |

![keys](/docs/getting-started/features/images/keys.webp)
Infrastructure credentials are **not** part of the Shoot itself — they are `Secret`s in the garden cluster's project namespace, referenced by Shoots via a `CredentialsBinding`. Shoot credentials are generated per Shoot by Gardener and rotate through well-defined phases.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Clarify user versus Gardener responsibilities.

“Who rotates: You” and “You, via kubectl annotate” can imply that users perform the rotation mechanics. State explicitly that users choose the timing, update dependent clients, and trigger completion, while Gardener creates, distributes, and invalidates Gardener-managed credentials. Add a short decision guide for choosing between the two workflows.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 20 - 25, Update the credential responsibility table and surrounding
explanation to distinguish user actions from Gardener-managed rotation
mechanics: users choose timing, update dependent clients, and trigger
completion, while Gardener creates, distributes, and invalidates Shoot
credentials. Add a concise decision guide explaining when to use the
infrastructure-credentials workflow versus the Shoot-credentials workflow,
anchored to the existing “Infrastructure credentials” and “Shoot credentials”
sections.

Comment on lines +31 to +43
## Two-Phase Rotation Model

Where possible, the rotation happens in two phases - Preparing and Completing.
For Gardener-managed credentials, rotation happens in two phases where possible.

### Prepare Rotation of All Credentials
![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)

The Preparing phase introduces new keys while the old ones are still valid.
Users can safely exchange keys / CA bundles wherever they are used.
It is possible to start the preparation by annotating the shoot resource accordingly:
In the **Preparing phase**, new credentials are created alongside the old ones — both sets are valid simultaneously.
This gives you time to update any API clients, kubeconfigs, or tooling that depend on the old credentials before they are invalidated.

```bash
kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-start
```
In the **Completing phase**, the old credentials are invalidated and only the new set remains.
You should only trigger this phase after all clients have been updated to use the new credentials.

### Complete Rotation of All Credentials
The shoot's status always reflects the current rotation phase, readable at `.status.credentials.rotation`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add readiness checks and failure consequences to the two-phase workflow.

The page tells users to update clients before completing rotation, but does not identify how to verify readiness, what happens if a client is missed, or how to recover from a failed completion. This is especially risky for CA and kubeconfig consumers, where premature completion can break access.

Suggested addition
 In the **Completing phase**, the old credentials are invalidated and only the new set remains.
 You should only trigger this phase after all clients have been updated to use the new credentials.
+Before completing, verify every affected client and confirm the rotation status is ready.
+If any client still uses the old credentials, it may lose access after completion; document the
+recovery procedure and whether another rotation is required.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Two-Phase Rotation Model
Where possible, the rotation happens in two phases - Preparing and Completing.
For Gardener-managed credentials, rotation happens in two phases where possible.
### Prepare Rotation of All Credentials
![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)
The Preparing phase introduces new keys while the old ones are still valid.
Users can safely exchange keys / CA bundles wherever they are used.
It is possible to start the preparation by annotating the shoot resource accordingly:
In the **Preparing phase**, new credentials are created alongside the old ones — both sets are valid simultaneously.
This gives you time to update any API clients, kubeconfigs, or tooling that depend on the old credentials before they are invalidated.
```bash
kubectl -n <shoot-namespace> annotate shoot <shoot-name> gardener.cloud/operation=rotate-credentials-start
```
In the **Completing phase**, the old credentials are invalidated and only the new set remains.
You should only trigger this phase after all clients have been updated to use the new credentials.
### Complete Rotation of All Credentials
The shoot's status always reflects the current rotation phase, readable at `.status.credentials.rotation`.
## Two-Phase Rotation Model
For Gardener-managed credentials, rotation happens in two phases where possible.
![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)
In the **Preparing phase**, new credentials are created alongside the old ones — both sets are valid simultaneously.
This gives you time to update any API clients, kubeconfigs, or tooling that depend on the old credentials before they are invalidated.
In the **Completing phase**, the old credentials are invalidated and only the new set remains.
You should only trigger this phase after all clients have been updated to use the new credentials.
Before completing, verify every affected client and confirm the rotation status is ready.
If any client still uses the old credentials, it may lose access after completion; document the recovery procedure and whether another rotation is required.
The shoot's status always reflects the current rotation phase, readable at `.status.credentials.rotation`.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 31 - 43, Expand the “Two-Phase Rotation Model” section with concrete
readiness checks before completing rotation, including how to verify all CA and
kubeconfig consumers use the new credentials. Describe the access failures
caused by completing with a missed client and document the recovery path for a
failed completion, using the existing rotation status reference where
applicable.

Comment on lines +49 to +57
## Automatic Rotation

![rotation-phases](/docs/getting-started/features/images/rotation-phases.webp)
Some Gardener-managed credential types support automatic rotation during the maintenance window via `.spec.maintenance.autoRotation.credentials`:

At the beginning, only the old set of credentials exists.
By triggering the rotation, new credentials are created in the Preparing phase and both sets are valid.
Now, all clients have to update and start using the new credentials.
Only afterward it is safe to trigger the Completing phase, which invalidates the old credentials.
- SSH key pair
- ETCD encryption key (enabled by default on new shoots)
- Observability passwords

The shoot's status will always show the current status / phase of the rotation.
Certificate authorities and the ServiceAccount signing key require user action between phases and therefore cannot be rotated automatically.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document credential types that cannot be rotated.

The automatic-rotation list omits the OpenVPN TLS auth key, which the detailed guide identifies as not rotatable manually or automatically. Explicitly distinguish manual-only credentials from credentials that cannot be rotated at all, so users do not search for or attempt an unavailable procedure.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 49 - 57, Update the “Automatic Rotation” section to explicitly list the
OpenVPN TLS auth key as a credential that cannot be rotated manually or
automatically. Distinguish it from certificate authorities and the
ServiceAccount signing key, which require manual rotation between phases, and
preserve the existing automatic-rotation list.

Comment on lines +61 to +68
## Infrastructure Credentials

![user-provided-keys](/docs/getting-started/features/images/user-provided-keys.webp)

You grant Gardener permissions to create resources by handing over cloud provider keys.
These keys are stored in a secret and referenced to a shoot via a SecretBinding.
Gardener uses the keys to create the network for the cluster resources, routes, VMs, disks, and IP addresses.
Infrastructure credentials are cloud provider keys you supply to Gardener so it can manage your cluster's infrastructure (networks, VMs, disks, load balancers).
These keys are stored in a `Secret` in the garden cluster's project namespace and referenced by your Shoot via a `CredentialsBinding`. A single `Secret` can be shared across multiple Shoots.

When you rotate credentials, the new keys have to be stored in the same secret and the shoot needs to reconcile successfully to ensure the replication to every controller.
Afterward, the old keys can be deleted safely from Gardener's perspective.

While the reconciliation can be triggered manually, there is no need for it (if you're not in a hurry).
Each shoot reconciles once within 24h and the new keys will be picked up during the next maintenance window.
When you rotate these credentials, you update the `Secret` with new keys, wait for all Shoots referencing that `Secret` to reconcile successfully, and only then deactivate the old keys in your cloud provider account.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add verification and failure handling for shared infrastructure credentials.

The instruction to “wait for all Shoots” is not actionable enough. Provide the expected reconciliation/status checks, timing expectations, and a procedure for handling one Shoot that fails to reconcile. Because the Secret may be shared, deactivating old cloud keys prematurely can disrupt multiple Shoots.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hugo/content/docs/getting-started/features/credential-rotation.md` around
lines 61 - 68, Expand the credential-rotation guidance around “When you rotate
these credentials” with actionable verification: specify how to confirm every
referencing Shoot has reconciled successfully, the expected timing or polling
interval, and the status conditions to check. Add failure handling for an
individual Shoot, including investigating or remediating it and keeping the old
provider keys active until all shared-Shoot reconciliations succeed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. kind/enhancement Enhancement, improvement, extension size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants